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Abstract 


Sequential  behavioral  data,  be  it  verbal  protocols,  automatically-tecorded  keystrokes,  or 
complete  videotape  protocols,  can  be  an^yzed  at  different  levels  of  detail  and  from 
different  viewpoints.  If  raw  behavioral  data  is  stored  in  a  powerful  database,  rather  than  a 
simple  text  file,  many  domains  will  aUow  some  automatic  interpretation  of  that  data.  In 
addition,  the  raw  data  can  be  compared  with  traces  of  an  associated  computational 
cognitive  model  to  assess  how  well  the  model  accounts  for  the  data  and,  conversely,  how 
much  support  the  behavioral  data  provides  for  the  components  of  the  model.  This  report 
describes  a  prototype  database  and  user  interface,  called  Trace&Transcription,  that  is 
designed  to  facilitate  protocol  analysis  and  o^nitive  modeling. 


This  research  was  supported  in  part  by  a  grant  from  U  S  West  Advanced  Technologies 
Sponsored  Research  Ingram.  The  views  and  conclusions  contained  in  this  document  are 
those  of  the  author  and  should  not  be  interpreted  as  representing  the  official  policies,  either 
expressed  or  implied,  of  U  S  West. 
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1.  Introduction 


Sequential  behavioral  data,  be  it  verbal  protocols,  automatically-recorded  keystrokes,  or 
complete  videotape  protocols,  can  be  analyzed  at  different  levels  of  detail  and  from  different 
viewpoints.  Behavior  is  often  first  described  as  observation  of  overt  action:  verbal 
utterances,  keystrokes,  mouse  movements,  screen  changes,  etc.  This  level  of  description  is 
pure  observation,  with  no  interpretation  added  by  the  an^yst,  and  is  often  compiled  either 
automatically  (e.g.,  keystroke  collection  programs)  or  by  clerical  personnel  skilled  in 
transcription.  This  description  produces  vast  amounts  of  data.  Sheer  quantity  makes 
handling  these  data  a  difficult  process.  A  long  standing  goal  of  researchers  who  use  this  type 
of  data  has  been  to  create  a  methodology  for  handling  its  quantity  and  complexity,  but  as  yet, 
no  single  method  has  emerged  as  a  standard  in  the  community.  Our  approach  is  to  set  up  a 
database  that  allows  us  to  relate  several  different  descriptions  of  the  same  behavior  in  an 
attempt  to  meet  this  goal. 

If  raw  behavioral  data  is  stored  in  a  powerful  database,  rather  than  a  simple  text  file,  many 
domains  will  allow  some  automatic  interpretation  of  that  data.  For  instance,  in  studying  a 
programming  environment  with  a  programming  window,  an  execution  window  and  a  help 
window  (e.g..  Figures  1  and  2),  a  powerful  database  would  be  able  to  identify  activity  in 
specific  windows  and  segment  the  behavior  into  broad  episodes.  That  is,  mouse  activity  in 
the  help  window,  probably  indicates  a  browsing  episode  where  the  user  is  looking  for  syntax 
or  semantics  of  a  command;  mouse  or  keyboard  activity  in  the  programming  window 
probably  indicates  a  code-edting  episode.  These  episodes  might  be  further  categorized  by 
their  length,  or  by  what  episodes  they  follow  or  precede.  For  instance,  long  segments  of 
behavior  with  verbal  utterances  but  no  mouse  or  keyboard  activity  might  indicate  either  a 
program-planning  episode  or  a  reading  episode,  depending  on  whether  they  are  preceded  and 
followed  by  programming  episodes  (for  planning)  or  browsing  episodes  (for  reading).  This 
second  level  of  analysis,  automatically  generated  by  the  database  itself,  could  also  be  stored 
in  the  database  and  dlow  the  analyst  to  easily  identify  the  behavior  for  more  detailed  study. 

Protocols  can  also  be  coded  for  critical  incidents,  events  where  the  user  either  makes  a 
mistake,  has  trouble  understanding  or  using  the  interface,  or  makes  some  explicit  remark 
either  praising  or  condemning  the  system.  These  critical  incidents  could  also  appear  in  the 
database. 

Another  frequently  used  method  of  analyzing  protocol  data,  is  to  code  the  behavior  according 
to  a  general  model  of  an  activity.  For  instance,  in  our  work  with  browsing,  we  have 
developed  a  model  that  decomposes  a  browsing  activity  into  three  sub-activities:  defining 
criteria  for  search  and  evaluation,  performing  ^  actual  search,  and  evaluating  the  results  of 
the  search.  The  protocol  data  automatically  identified  as  browsing,  could  then  be  examined 
and  hand-coded  for  these  sub-activities,  lliis  information  could  Aen  be  added  to  the 
database. 

Another  way  of  looking  at  the  data  might  be  to  perform  a  GOMS  analysis  of  segments  of 
behavior  (Card,  Moran  &  Newell,  1983).  The  raw  data  is  arranged  chronologically  and  the 
GOMS  hierarchy  of  goals  could  also  be  associated  with  segments  of  behavior  in  the 
database.  This  would  allow  identification  of  when  a  goal  is  begun,  suspended,  re-activated, 
and  completed.  Patterns  in  goal-switching  could  be  identified  and  quantified.  Both 
chronological  and  goal-hierarchic  displays  of  behavior  could  be  produced. 

Finally,  the  most  detailed  level  of  analysis  of  protocol  data  usually  undertaken  is  a  computer 
simulation  of  the  behavior.  For  instance,  we  have  a  cognitive  model  of  browsing  that 
produces  much  of  the  verbal  utterances  and  mouse  events  for  several  segments  of  browsing 
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behavior  (John,  Newell  &  Card,  1990;  Peck  &  John,  1992).  A  powerful  database  would 
allow  traces  of  a  computer  simulation  to  be  associated  with  individual  behavioral  events,  and 
facilitate  examination  of  the  goodness  of  fit  of  the  computer  simulation  to  the  behavior. 

Also,  correspondence  between  the  different  analysis  viewpoints  could  be  obtained,  and 
comparison  of  their  effectiveness  could  be  performed. 

2.  The  Data 

For  a  prototype  database,  we  chose  to  included  the  raw  videot£q)e  protocol  data  and  the  most 
detail^  level  of  analysis,  a  trace  of  a  cognitive  simulation  model.  The  prototype 
demonstrates  how  these  two  levels  of  analysis,  one  the  pure  description  of  observed 
behavior  and  the  other  a  generalized  model  of  that  behavior,  could  be  related  usefully  in  a 
single  database  system. 

2.1  Protocol  Data 

We  focused  on  the  browser  associated  with  the  on-line  reference  manual  for  the  cT 
programming  language.^  cT  provides  a  highly  interactive  programming  environment  where, 
typically,  code  is  entered  into  one  window  (the  programming  window)  and  the  results  of  the 
code  are  displayed  in  a  second  window  (the  execution  window)  (Figure  1).  Changes  in  the 
code  produce  changes  in  the  display  and,  conversely,  a  g^hic  edtor  allows  changes  to  be 
made  directly  in  the  execution  window  which  are  imme^ately  reflected  in  the  code.  In 
addition  to  the  programming  and  execution  windows,  a  Help  window  can  be  brought  up  to 
provide  access  to  the  on-line  reference  manual  (an  exact  duplicate  of  the  hard-copy  reference 
manual).  This  Help  window,  shown  in  Figure  2,  is  the  browser  of  interest  in  our  research. 
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Figure  1.  The  typical  cT  programming  environment  on  the  Apple  Macintosh.  The  window  on  the 
left  is  the  programming  window  where  code  is  entered.  The  window  on  the  right  is  the  execution 
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window  where  the  results  of  the  code  are  displayed.  Changes  to  the  code  produce  changes  in  the 
execution  window  and  changes  made  with  a  graphic  editor  in  the  execution  window  produce 
immediate  changes  in  the  code. 


Figure  2.  The  cT  Help  window.  The  window  on  the  top  left  is  a  hierarchical  menu  similar  to  a 
table  of  contents  in  a  hard-cc^y  reference  manual.  The  window  on  the  top  right  is  an  alphabetical 
list  of  all  cT  commands.  The  bottom  window  displays  the  reference  material  for  the  commands. 


There  are  three  sub-windows  in  the  help  window:  a  hierarchical  menu  (top  left),  an 
alphabetic  list  of  cT  commands  (t(^  ri^t),  and  tire  help  text  (bottom).  The  two  top  windows 
provide  two  access  paths  to  the  ^ta,  which  is  displayed  in  the  bottom  window,  ^^^n  a  user 
needs  information  about  a  cT  command,  he  or  she  Wings  up  the  help  window  and  searches 
for  that  information  either  by  selecting  increasingly  more  specific  items  from  the  hierarchical 
iirenu  or  selecting  a  particular  conunand  the  alph^tical  list.  When  the  user  selects  items 
from  the  hierarctucal  menu,  the  menu  changes  to  increasingly  more  specific  items  until 
specific,  related  commands  are  displayed  in  the  menu.  Wlren  the  user  selects  a  specific 
command  in  the  hierarchical  menu,  that  comnumd  is  highlighted  in  bold  print  and  the 
reference  text  for  that  command  t^rpears  in  the  help  text  window.  Selections  in  the 
hierarchical  menu  do  not  change  toe  alphabetic  list.  However,  when  a  user  selects  a 
command  in  toe  alphabetic  list,  several  things  luqrpen.  First,  toe  selected  conunand  is 
highlighted  in  bold  and  the  alphabetic  list  is  scroll^  up  so  toe  selected  command  is  at  toe  top 
of  toe  list.  Next  the  reference  material  for  that  conunand  is  displayed  in  toe  help  text 
window.  Last,  toe  hierarchical  menu  is  changed  so  that  toe  selected  conunand  (highlighted 
in  bold),  and  commands  related  to  it,  are  displayed  in  the  hierarchical  menu. 


A  stated  purpose  of  cT  is  to  allow  non-professional  programmers  to  easily  create  highly 
interactive  graphic  q>plications.  A  volunteer,  non-professional  but  experienced  progranuner 
(conversant  in  USP,  Pascal,  FORTRAN,  Basic,  and  assembly  language)  needed  to  create  a 
program  to  read  psychological  data  from  a  spreadsheet  file  and  plot  it  on  x-y  coordinates. 
Iliis  person  Oiereafier  called  Ul)  had  never  used  cT,  but  was  willing  to  spend  an  afternoon 
attempting  to  create  her  application  with  cT,  while  being  videotaped.  She  was  asked  to  use 
only  ^  on-line  reference  manual  when  she  needed  information  about  cT  in  toe  course  of  her 
progranuning.  Three  and  a  half  hours  of  programming  were  videotaped,  during  which  time 
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U1  completed  the  graphing  part  of  her  application.  An  additional  four  hours,  not  videotaped, 
was  needed  to  complete  the  ihle  input  portion  of  the  application. 

We  identified  three  types  of  behavior  in  the  videotaped  segment;  the  initial  reading  for  general 
information,  coding,  and  browsing  for  specific  information  needed  in  the  course  of  coding. 
Of  the  3.5  hours,  0.45  hours  were  spent  on  the  initial  reading,  1 .80  hours  were  spent 
coding,  and  1 .25  hours  were  spent  browsing.  These  1 .25  hours  of  browsing  represent  80 
browsing  sequences  averaging  56  seconds  apiece  with  a  range  of  1  to  270  seconds. 

To  date,  all  of  the  browsing  sequences  have  been  transcribed  so  that  every  verbal  utterance, 
every  mouse-movement  and  button  click,  and  every  keystroke  is  recorded  in  a  format 
automatically  readable  by  the  eventual  database  system. 

2J2  Trace  Data 

A  single,  67  second  browsing  sequence  was  chosen  for  detailed  examination  and  preliminary 
modeling.  Before  this  browsing  sequence,  U1  had  produced  code  that  displayed  labeled 
axes  on  the  screen  and  she  now  needed  to  place  specific  data  points  on  the  graph.  She  was 
looking  for  a  cT  command  that  would  produce  a  "filled  circle" .  The  transcription  of  this 
browsing  segment  appears  in  Figure  3.  We  used  this  transcription  as  a  guide  for  building  a 
preliminary  model  of  this  browsing  incident. 

The  cognitive  model  was  built  on  the  Soar  architecture  of  cognition.  As  succinctly  described 
in  Lewis,  et.  ai,  (1990)  and  described  in  more  detail  elsewhere  (Newell,  1990)  the  Soar 
architecture  formulates  all  tasks  in  problem  spaces,  in  which  operators  are  selectively  applied 
to  the  current  state  to  attain  desired  states.  I^blem  solving  proceeds  in  a  s^uence  of 
decision  cycles  that  select  problem  spaces,  states  and  operators.  Each  decision  cycle 
accumulates  knowledge  from  a  long  term  recognition  memory  (realized  as  a  production 
system).  This  memory  continually  matches  against  working  memory,  elaborating  the  current 
state  and  retrieving  preferences  that  encode  knowledge  about  the  next  step  to  take.  Access  of 
recognition  memory  is  involuntary,  parallel,  and  rtq>id  (assumed  to  take  on  the  order  of  10 
milliseconds).  The  decision  cycle  accesses  recognition  memory  repeatedly  to  quiescence,  so 
each  decision  cycle  takes  on  the  order  of  100  milliseconds. 

If  Soar  does  not  know  how  to  proceed  in  a  problem  space,  an  impasse  occur.  Soars 
responds  to  an  impasse  by  creating  a  subgoal  in  which  a  new  problem  space  can  be  used  to 
acquire  the  needed  knowledge.  If  lack  of  knowledge  prevents  progress  in  a  new  space, 
another  subgoal  is  created  and  so  on,  creating  a  go^-subgoal  hierarchy.  Once  an  impasse  is 
resolved  by  problem  solving,  the  chunking  mechanism  adds  new  productions  to  recognition 
memory  encoding  the  results  of  the  problem  solving,  so  the  impasse  is  avoided  in  the  future. 
All  incoming  perception  and  outgoing  motor  commands  flow  through  the  state  in  the  top 
problem  space. 

Figure  4  shows  the  structure  of  Soar  when  it  is  functioning  in  a  highly  interactive 
environment.  The  decision  cycle  is  unchanged;  aU  available  information  is  accumulated 
about  the  acceptability  and  desirability  of  problem  spaces,  states,  and  operators  for  the  total 
current  context,  and  the  best  alternative  is  chosen  among  those  that  are  acceptable.  However, 
in  a  highly  interactive  environment,  the  outside  world  (in  this  case,  the  cT  display  on  the 
CRT  screen)  is  an  important  source  of  the  information  being  collected  in  working  memory. 
This  influx  of  information  influences  the  decision  cycle  both  by  directly  depositing  relevant 
information  into  working  memory  and  by  triggering  long-term  memory  to  deposit  other 
general  knowledge  into  working  memory.  The  outcome  of  each  decision  cycle  is  to  select  a 
problem  space,  state,  or  operator  for  application,  or  generate  an  impasse.  In  a  highly 
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Screen  Changes 
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OK 

06 

so 

08 

so  1  just  want  to 

13 

makeamark 

15 

1  want  to  draw  something 

16 

1  w»it  to  drde  something 

18 

guess  1  want  to  put  a  little  circle  in 

19 

M  to  help  screen 

20 

so,  let's  see 

C  on  help  screen 

help  whxlow  comes  up 

21 

1  want  a  filled  circle 

22 

let's  find... 

M  to  command  menu 
D  on  up  scroll  arrow 

command  menu  scrolls  up 

23 

corrvnand  menu 

gdrde  appears  in  the  scrolling 

24 

•geirde" 
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gdrde  scrolls  off  the  top 
command  menu  stops  scrolling 

25 

C  on  davn  arrow 

command  menu  pages  down 
gdrde  is  not  yet  in  the  window 

Con  down  arrow 

command  menu  pages  down 
gdrde  is  on  2nd  line  of  menu 

26 

"geirde" 

Mtogdrde-f  C 

gdrde  scrolls  to  top  of  menu,  turns  bold 
gdrde  help  text  comes  up 
hierarchical  menu  changes 
gdisk  is  visible  below  gdrde 

29 

1  think  it's  "gdisk" 

30 

actually 

M  to  gdisk 

31 

"gdisk" 

Con  gdisk 

gdisk  turns  bold,  help  text  comes  up 
(same  as  gdrde) 
hierarchical  menu  changes 

32 

gorma  give  me  the  same  thing 

33 

OK-so  1  really  want  to  look  at.., 

M  to  above  elevator 

34 

1  wonder  what  "doT  is 

35 

C  above  devator 

command  menu  pages  up 

36 

C  above  elevator 

oommarKi  menu  pages  up 

37 

what's  "dor? 

Mtodd-t-C 

dot  scrolls  to  top  of  menu,  turns  bold 
dot  help  text  comes  up 
hierarchical  menu  changes 

39 

making  dots  (start  reading) 

40 

single  dot 

41 

umm 

42 

OK 

45 

(reading)  the  dot  command 

46 

with  one  point  is 

47 

equivalent  to  a  draw  command 

48 

with  a  single  position  (end  readng) 

49 

oh  those  are  very  little  teeny 

50 

1  doni  want  those-l  want  di^ 

51 

M  around  command  menu 

52 

where's  disk? 

M  to  hierarchical  menu 

53 

dot  drde 

54 

draw  disk 

55 

fined  in  drde 

Mtodisk  +  C 

disk  turns  bold,  help  text  comes  up 

56 

disk  command 

57 

fined  in  drde 

58 

umm 

60 

disks,  OK 

Figure  3.  Transcription  of  the  videotape  protocol  segment  used  to  construct  the  preliminary  Soar 
model  of  browsing.  In  the  Mouse  Behavior  column,  M  -  move  mouse  so  cursor  moves,  D  s  press 
down  on  the  mouse  button,  U  =  release  die  mouse  button,  C  s  click  the  mouse  button  (rapid  down 
and  op).  Behavior  in  this  protocol  is  referred  to  by  its  timestamp,  e.g.,  t5S  refers  to  U1  saying 
"filled  in  circle”,  moving  the  mouse  to  pot  the  cursor  ion  the  word  "disk"  and  clicking  the  mouse. 
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interactive  environment,  the  application  of  operators  often  leads  to  changes  in  the  external 
environment  (in  this  case,  moving  the  mouse  and  pressing  its  button). 

The  preliminary  Soar  model  of  browsing  for  the  cT  coding  task,  Browser-Soar,  uses  the 
Soar  problem  space  structure  to  provide  the  goals,  operators,  methods  and  selection  rules 
found  in  traditional  GOMS  analyses.  Browser-Soar  consists  of  a  set  of  problem  spaces 
(Figure  5)  that  provide  the  capability  to  search  deliberately  through  the  help  windows,  while 
blowing  recognition  of  new  items  to  trigger  knowledge  at  any  time  that  may  change  the 
search  strategy.  The  top  problem  space  in  Browser-Soar  (Browse)  is  entered  when  an 
impasse  arises  in  the  task  space  for  programming  in  the  cT  language.  A  browsing  episode 
involves  bringing  up  the  help  window,  finding  the  appropriate  help,  and  applying  the  newly 
found  information  to  the  problem  at  hand.  Each  of  diese  activities  correspond  to  an  operator 
in  the  Browse  space.  Currently,  Browser-Soar  implements  the  find-appropriate>help 
operator.  Applying  this  operator  results  in  an  impasse  because  the  operator  cannot  be 
implemented  by  recognition.  Soar  responds  by  setting  up  another  problem  space,  with 
operators  that  define  de  search  criteria  (e.g.,  what  labels  to  look  for  in  the  help  windows), 
define  the  evaluation  criteria  (how  to  decide  that  some  piece  of  information  will  actually  help 
resolve  the  impasse  in  the  task  space),  carry  out  the  search,  and  evaluate  the  search  results. 
Each  of  these  operators  is  also  implemented  in  a  problem  space;  for  example,  carrying  out  the 
search  for  the  defined  criteria  is  accomplished  in  a  space  with  operators  that  select  among 
search  methods  and  execute  them.  At  the  bottom  of  the  problem  space  hierarchy  are  motor 
operators  that  control  mouse  and  keyboard  actions,  and  cognitive  operators  that  can  be 
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Figure  4.  Soar  in  a  highly  interactive  environment. 
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applied  with  directly  available  knowledge.  The  operators  in  Browser-Soar  can  be  viewed  as 
deliberate  goals,  and  this  organization  is  useful  for  modeling  the  goal-oriented  component  of 
browsing  as  well  as  the  mechanics  of  manipulating  the  windows  used  for  browsing.  Data- 
driven,  opportunistic  behavior  emerges  because  an  additional  operator,  evaluate-new* 
items,  is  proposed  whenever  new  information  is  brought  within  the  scope  of  attention 
evaluate-new-items  is  available  in  every  Browser-Soar  problem  space).  The  current 
problem  solving  is  thus  interrupted  so  the  new  items  may  be  considered,  possibly  suggesting 
a  more  relevant  path  to  pursue. 

When  given  the  necessary  knowledge  about  the  user's  goals  and  the  information  visible  on 
the  screen  in  the  protocol  of  Figure  3  and  the  problem  space  structure  in  Figure  5,  Browser- 
Soar  produces  the  trace  in  Figure  6. 
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Figure  6.  Browser-Soar  trace  (condnues  for  3  pages) 
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G:  gl 

P:  p4  (top -space) 

S:  s5 

O:  ol2  (browsing- task) 

==>G:  g36  (operator  no-change) 

P :  p42  ( brows ing ) 

S:  s49 

O:  o51  (find-appropriate-help) 

==>G:  g53  (operator  no-change) 

P:  p59  (find-appropriate-help) 

S:  s72 

O:  o74  (define-search-criterion) 

==>G:  g78  (operator  no-change) 

P:  p84  (define-search-criterion) 

S:  s94 

O:  olOO  (generate-search-criterion) 

O:  oll2  (evaluate-search-criterion) 

O:  oll8  (generate-search-criterion) 

0:  0128  (evaluate-search-criterion) 

O;  ol34  (generate-search-criterion) 

O:  ol42  (evaluate-search-criterion) 

0:  o93  (final-state) 

O:  o76  (def ine-evaluation-criterion) 

==>G:  gl48  (operator  no-change) 

P:  pl54  (def ine-evaluation-criterion) 

S;  sl65 

0;  ol69  (generate-evaluation-criterion) 

O:  ol77  (evaluate-evaluation-criterion) 

O:  ol64  (final-state) 

O:  ol85  (search-for-criterion) 

==>G:  gl87  (operator  no-ch6uige) 

P:  pl93  (search-for-criterion) 

S:  s212 

O;  o214  (select-search-method) 

==>G:  g216  (operator  no-change) 

P:  p222  (select-search-method) 

S:  s236 

O:  o238  (choose -window) 

O;  o244  ( focus-on-current -window) 

O:  o255  (evaluate-current -window) 

O:  o289  (choose-method) 

O:  o295  (choose-direction) 

O:  o235  (final-state) 

O:  o311  (execute-search-method) 

==>G:  g313  (operator  no-change) 

P:  p319  (execute-search-method) 

S:  s335 

O:  o337  (move-mouse-to-up-arrow) 

0:  o342  (press-mouse-button-scroll-up) 
O;  o351  (perceive-search-criterion) 

O:  o349  (release-mouse-button) 

O:  o371  (evaluate-new-items-nn) 

0:  o362  (get-location-of-coranand) 

==>G:  g375  (operator  no-chemge) 

P:  p381  (back-up) 
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S:  s394 

O:  o396  (change-direction) 

O;  o398  (chauige-method) 

O:  o393  (final-state) 

O:  o416  (move-mouse-to-below-elevator) 
0:  o421  (click-mouse-button) 

O:  o438  (evaluate-new-items-nn) 

O:  o421  (click-mouse-button) 

O:  o453  (perceive-search-criterion) 

O:  o459  (evaluate-new-items-nn) 

O:  o463  (get-location-of-command) 

O:  o334  (final-state) 

O:  o480  (bring-up-help-text) 

==>G:  g482  (operator  no-change) 

P:  p488  (bring-up-help-text) 

S;  s499 

O:  o501  (move-mouse-to-item-location) 
O:  o506  (click-mouse-button) 

O:  o498  (final-state) 

O:  o544  (evaluate-new-i terns) 

0:  o558  (bring-up-help-text) 

==>G:  g560  (operator  no -change) 

P:  p566  (bring-up-help-text) 

S:  s577 

O;  o579  (move-mouse-to-item-location) 
O:  o584  (click-mouse-loutton) 

O:  o576  (final-state) 

O:  o606  (evaluate-new-items-nn) 

0:  o211  (final-state) 

O:  o615  (evaluate-help-text) 

==>G:  g617  (operator  no-ch<ange) 

P:  p623  (evaluate-help-text) 

S:  s632 

0:  o634  (focus-on-help-text) 

O:  o638  (find-pointer-to-criterion) 

O:  o631  (final-state) 

O:  o655  (search-for-criterion) 

==>G:  g657  (operator  no-change) 

P:  p663  (search-for-criterion) 

S:  s682 

O:  o684  (select-search-method) 

==>G:  g686  (operator  no-change) 

P:  p692  (select-search-roethod) 

S:  s706 

O:  o708  (choose-window) 

O;  o716  ( focus-on-current -window) 

O:  o721  (evaluate-current -window) 

O:  o755  (choose-method) 

O:  o761  (choose-direction) 

O:  o705  (final-state) 

O:  o777  (execute-search-method) 

==>G:  g779  (operator  no-change) 

P:  p785  (execute-search-raethod) 

S:  s801 

O:  o803  (move-mouse- to-above-elevator) 
O;  o808  (click-mouse-button) 

O:  o825  (evaluate-new-items-nn) 

O:  o808  (click-mouse-button) 
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113 

0:  o840  (perceive-search-criterion) 

114 

0:  o846  (evaluate-new-i terns -nn) 

115 

0:  o850  (get-location-of-coimnand) 

116 

0:  o800  (final-state) 

117 

0:  o867  (bring-up-help-text) 

118 

==>G:  g869  (operator  no-change) 

119 

P:  p875  (bring-up-help-text) 

120 

S;  s886 

121 

0:  o888  (move-mouse-to-item-location) 

122 

0:  o893  (cliclc-mouse-button) 

123 

0:  o885  (final-state) 

124 

0:  o915  (evaluate-new-items-nn) 

125 

0:  o681  (final-state) 

126 

0:  o924  (evaiuate-help-text) 

127 

==>G:  g926  (operator  no-chamge) 

128 

P:  p932  (evaluate-help-text) 

129 

S:  s941 

130 

0:  o943  ( focus-on-help-text) 

131 

0:  o949  ( find-disconfirmation) 

132 

0:  o940  (final-state) 

133 

0:  o972  ( search- for-criterion) 

134 

==>G:  g974  (operator  no-change) 

135 

P:  p980  (search- for-criterion) 

136 

S;  s999 

137 

0:  olOOl  (select-search-method) 

138 

==>G:  gl003  (operator  no-change) 

139 

P:  pl009  (select-search-method) 

140 

S:  S1023 

141 

0:  ol025  (choose-window) 

142 

0:  ol031  (focus-on-current-window> 

143 

0:  ol036  (evaluate-current-window) 

144 

0:  ol022  (final-state) 

145 

0;  oll26  (bring-up-help-text) 

146 

==>G:  gl038  (operator  no-change) 

147 

P:  pi 12 8  (bring-up-help-text) 

148 

S;  sl045 

149 

0:  oll39  (move-mouse-to-item-location) 

150 

0:  01144  (click-mouse-button) 

151 

0:  01137  (final-state) 

152 

0:  01166  (evaluate-new-items-nn) 

153 

0:  o998  (final-state) 

154 

0:  01175  (evaluate-help-text) 

155 

==>G;  gl050  (operator  no-change) 

156 

P:  pll77  (evaluate-help-text) 

157 

S:  S1052 

158 

0:  oil86  (focus-on-help- text) 

159 

0:  oll90  (find-criterion) 

160 

0:  01184  (final-state) 

161 

0:  o71  (final-state) 

162 

0: 

o48  (final-state) 

163  0: 

o6 

(halt) 

(end  of  Figure  6.  Browser-Soar  trace) 
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3.  The  Database 

As  stated  previously,  the  goals  of  the  database  are  to  relate  several  levels  of  protocol 
analysis,  from  raw  protocol  transcription  data  through  the  most  detailed  cognitive  simulation 
traces,  and  to  allow  complex  database  manipulation  to  perform  automatic  classification  of  the 
data.  The  prototype  database  relates  a  raw  protocol  transcription  of  a  segment  of  browsing 
behavior  (Figure  3)  to  a  trace  of  the  Soar  model  of  that  behavior  (Figure  6). 

The  prototype  database  application,  called  Trace«&Transcription,  was  built  in  the  Oracle 
Relational  Database  Management  System  We  chose  Oracle  because  it  seemed  to  have  the 
best  mix  of  power  and  performance  with  ease-of-use  and  standardization  of  the  relational 
databases.  Oracle  can  be  used  either  as  a  stand-alone  application  on  a  personal  workstation 
(the  prototype  works  on  an  Apple  Macintosh),  or  as  a  multi-user  database  on  a  central 
processor  with  individual  access  through  networked  workstations.  It  provides  the  power  of 
the  industry  standard  query  language,  SQL.  On  the  Macintosh,  Oracle  provides  a 
HyperCard^  interface  for  loading  data  in  many  formats  (SQL* Loader  Utility),  constmcting 
queries  (Query  Tool),  and  creating  other  HyperCard  application  interfaces  (Application 
Generator)  (see  Oracle  for  Macintosh  User's  Guide,  Version  1.2,  (Dracle  Corp,  1990). 
Trace&Transcription  was  built  using  HyperCard  and  Oracle’s  HyperCard  interface  to  the 
database. 

3.1  Organization  of  the  Database 

The  database  has  one  table  for  the  raw  protocol  transcription  data  and  one  additional  table  for 
each  level  of  analysis  performed  on  the  protocol  data;  in  the  case  of  the  prototype  database, 
there  is  only  one  additional  table  for  Soar  traces.  These  analysis  tables  can  be  created  by 
hand  by  the  analyst,  or  they  can  be  created  automatically  by  sophisticated  query  techniques 
provided  by  Oracle.  This  prototype  uses  only  hand-created  analysis  tables. 

Each  record  in  the  protocol  transcription  table  represents  a  one  second  time  period,  during 
which  a  verbal  utterance  has  occurred,  a  mouse  action  has  occurred  (move,  click,  press- 
button  or  release-button),  neither,  or  both  have  occurred.  Each  record  in  the  Soar  trace  table 
represents  a  single  decision  cycle  for  Browser-Soar  which  terminated  in  the  selection  of  a 
goal,  problem  space,  state,  or  operator.  A  useful  relationship  between  the  two  tables  is  to 
link  a  Soar  decision  cycle  to  the  behavior  that  is  direct  evidence  for  the  action  produced  by 
that  decision  cycle,  e.g.,  an  observed  mouse-button  click  would  be  direct  evidence  for  the 
selection  of  the  Soar  operator  click>inotise*button.  This  is  a  symmetric  relationship, 
where  the  observed  behavior  is  evidence  for  application  of  Soar  operators,  and  the  selection 
of  Soar  operators  provide  a  simulation  of  human  behavior.  Our  database  application, 
Trace&Transcription,  allows  a  user  to  explicitly  create  these  links  and  the  database  records 
them  by  creating  a  third  table  for  pointers  between  the  two  user-provided  tables. 


^Trademark  of  the  Oracle  Corporation. 

^HyperCard  is  a  trademark  of  Apple  Computer,  Inc. 


The  TRANSCRIPTION  table  has  six  columns: 


BEHAVIOR_NUMBER 

Type:  CHAR 

Size:  10 

Required?  no 

Description:  This  is  the  unique  number  of  the  single  behavior  in  one  line  of  the 

transcription  of  this  segment  of  behavior.  It  ends  with  the  character 
"v"  if  it  is  a  verbal  utterance,  or  "m"  if  it  is  motor  action  like  mouse 
movement  or  button  clicking. 


NUMBER 
12 
yes 

This  is  the  time  in  milliseconds  since  the  beginning  of  the  t£q)e.  This 
number  2q)pears  in  the  bottom  right-hand  comer  of  the  videot£q)e. 

Each  new  record  starts  one  second  (1000  msec)  after  the  record 
preceding  it. 

VERBAL 

Type:  CHAR 

Size:  100 

Required?  no 

Description:  This  is  the  verbal  utterance  that  happened  during  the  one  second  (1000 

msec)  after  the  TAPEJTIME  of  this  record,  if  any. 

POINTING 

Type;  CHAR 

Size;  100 

Required?  no 

De^ption:  This  is  the  motor  action  that  happened  during  the  one  second  ( 1000 
msec)  after  the  TAPEJTIME  of  this  record,  if  any. 

TRANSCRIPTION_ID 

Type:  CHAR 

Size:  10 

Required?  yes 

Description;  This  is  the  number  of  the  transcription  segment.  In  the  prototype 

database,  this  number  is  always  1  and  the  loading  program  provides 
this  constant. 

TAPE_ID 

Type;  CHAR 

Size;  10 

Required?  yes 

Description;  Hiis  is  the  number  of  the  videotape.  In  the  prototype  database,  this 

number  is  always  1  and  the  loading  program  provides  this  constant. 


TAPE_TIME 

Type: 

Size: 

Required? 

Description: 
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The  SOAR  table  has  four  columns: 

UNE.NUMBER 

Type:  NUMBER 

Size:  10 

Required?  yes 

Description:  This  is  the  unique  number  for  each  line  in  this  Soar  trace.  It  is 

provided  by  the  loading  program. 


DECISION.CYCLE 

Type:  NUMBER 

Size:  10 

Required?  yes 

Description:  This  is  the  decision  cycle  of  the  Soar  trace. 


TEXT 

Type: 

Size: 

Required? 

Description: 


CHAR 

200 

no 

This  is  the  text  of  the  Soar  trace, 
e.g.,  "O:  o342  (press-mouse-button)" 


SOAR_ID 

Type:  CHAR 

Size:  10 

Required?  yes 

Description:  This  is  the  number  of  the  Soar  trace.  If  several  variants  of  a  Soar 

model  are  being  evaluated,  there  may  be  multiple  traces  that  mq)  to  the 
same  protocol  data.  In  the  prototype  database,  this  number  is  always 
1  and  the  loading  program  provides  this  constant. 


The  POINTER  table  has  seven  colunms,  automatically  created  in  the  process  of  using 
Trace&Transcription  to  explicitly  provide  links  between  transcription  and  trace: 

TAPE.ID  -  taken  from  the  transcription  record  (required) 

TRANSCRIPTION_ID  -  taken  from  the  transcription  record  (required) 

TAPE_’nhffi  -  taken  from  the  transcription  reco^  (required) 

BEHAVIOR_NUMBER  -  taken  from  the  transcription  record  (^uired) 

(together  the  above  four  items  point  to  a  unique  transcription  record) 

SOAR_ID  -  taken  from  the  Soar  trace  record  (required) 

LINE_NUMBER  -  taken  from  the  Soar  trace  record  (required) 

(together  the  above  two  items  point  to  a  unique  Soar  trace  record) 

FORCED.AUGNMENT 

Type:  CHAR 

Size:  1 

Required?  yes 

Description:  This  is  either  "Y"  or  "N",  where  "Y"  means  that  the  unique 

transcription  record  should  be  spatially  aligned  with  the  unique  Soar 
trace  record  when  it  is  displayed  in  the  database  application,  and  "N" 
means  tlu  *  the  two  are  related  but  are  not  to  be  spatially  aligned  in  the 
display. 
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32  Operation  of  the  Database  Application  Relating 
Transcriptions  to  Traces 

This  section  describes  the  operation  of  the  Trace&Transcription  database  application  written 
for  Oracle;^  it  is  not  intended  as  an  instruction  manual  for  die  operation  of  the  Oracle 
Relational  Database  Management  System  itself.  Please  refer  to  the  Oracle  for  Macintosh 
User's  Guide  Version  1 .2  where  referenced  in  this  section.  We  assume  that  the  reader  is 
familiar  with  the  installation,  logon  procedure  and  use  of  Oracle  and  HyperCard. 

3,2.1  Loading  Protocol  and  Soar  Trace  Data 

The  protocol  transcription  data  and  Soar  traces  must  be  loaded  into  dieir  respective  t?'^ 
before  the  Trace&Transcription  application  can  be  used.  Before  loading  data,  the  ta 
described  above  must  be  created  in  the  Oracle  database  on  which  Trace&TranscriptiL  j 
be  run.  Refer  to  Oracle  User's  Guide  Chapter  2,  Managing  Tables  and  Views,  pages  .^.-9 
through  2-15,  for  instructions  for  creating  tables.  Then  refer  to  Chapter  2,  Loading  Data 
with  SQL*Loader,  pages  2-30  through  2-36,  for  instructions  for  loading  the  data.  The 
following  four  files  must  be  resident  on  the  host  computer  to  load  the  transcription  and  trace 
data: 


transcriptionxtl,  a  file  used  by  Oracle  to  control  the  loading  of  the  transcription  data.  The 
content  of  transcription  .ctl  is  shown  in  Figure  7.  The  INFT^  line  must  be  edited  to 
conform  to  the  names  of  the  disk  and  file  structure  of  the  host  computer. 


traiiscriptioii.dat,  a  text  file  with  the  transcription  data  in  the  format  described  by 
transcription  .ctl.  An  example  of  the  protocol  transcription  data  is  shown  in  Figure  8. 

soarxtl,  a  file  used  by  Oracle  to  control  the  lotuling  of  the  Soar  trace  data.  The  content  of 
soar  .ctl  is  shown  in  Figure  9.  The  INFILE  line  must  be  edited  to  conform  to  the  names 
of  the  disk  and  file  structure  of  the  host  computer. 

soarilat,  a  text  file  with  the  Soar  trace  in  the  format  described  by  soar  .ctl.  An  example  of 
the  Soar  trace  data  is  shown  in  Figure  10. 


LOAD  DATA 

INFILE  "<disk  name>:<foIdei>:<subfolder>:<filename>.dat'’  STREAM 


INTO  TABLE  transcription  replace 
FIELDS  TERMINATED  BY 
OPTIONALLY  ENCLOSED  BY 
( behavior_number  CHAR, 

tape_time  INTEGER  EXTERNAL, 

verbal  CHAR, 

pointing  CHAR, 

transcriptionjd  CONSTANT  1 ,  -  Constant  only  in  prototype 

tapejd  CONSTANT  1 

) 


Figure  7.  The  file  that  allows  Oracle  to  load  protocol  transcription  data,  transcription  .ctl 


^  At  the  initial  writing  of  this  report  in  the  winter  of  1990,  Trace&Transcription  was  available  as  a 
HyperCard  stack.  This  prototype  is  no  longer  available,  but  a  subsequent  system  has  been  built  upon  several 
of  the  interface  design  ideas  (Ritter,  1992;  Ritter  &  Larkin,  in  press). 
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V1,15705000,OK."" 

V2,15706000,so,"'’ 

,1 5707000 

V3,15708000^o  1  just  want  to,"" 

,15709000,,"" 

,15710000,,"" 

,15711000,,"" 

,15712000,,"" 

V4,157130004nake  a  mark,"" 

,15714000,,"" 

V5,15715000J  want  to  draw  something,"" 
V6,15716000 J  want  to  circle  something,"" 
,15717000,,"" 

V7,15718000,guess  I  want  to  put  a  little  circle  in,"" 
M8,15719000„"M  to  help  screen" 


Figure  8.  First  10  lines  of  the  protocol  transcription  data  file;  example  of  the  format 


LOAD  DATA 

INFILE  "<disk  name>:<foldei>:<subfolder>;<filename>.dat" 
INTO  TABLE  soar  replace 
FIELDS  TERMINATED  BY 
OPTIONALLY  ENCLOSED  BY 


(Line_Number 
Decision_Cycle 
Text 
Soar  id 
) 


recnum, 

INTEGER  EXTERNAL, 

CHAR(200), 

CONSTANT  1  --  Constant  only  in  prototype 


Figure  9.  The  file  that  allows  Oracle  to  load  Soar  trace  data,  soar.ctl 


0,  "G;  gl  " 

1 ,  "P:  ^  (top-space)" 

2,  "S:  s5  " 

3,  "O:  ol2  (browsing-task)" 

4,  "=>G:  g36  (operator  no-change)" 

5,  "  P:  p42  (browsing)" 

6,  "  S:  s49  " 

7,  "  O:  o51  (find-t^)propriale-help)" 

8,  "  =>G;  g53  (operator  no-change)" 

9,  "  P:  p59  (fmd-q>propriate-help)" 

10,  "  S: s72  " 


Figure  10.  First  10  lines  of  the  Soar  trace  data  file;  an  example  of  the  format 


16 


32.2  Running  the  Trace&Transcription  Application 

After  the  tables  are  created,  the  data  are  loaded,  and  Oracle  is  running,  Trace&Transcription 
can  be  run.  To  start  Trace&Transcription  from  Oracle's  Log  On  card,  click  on  the  Hmne 
icon,  and  enter  Trace&Transcription  through  the  dialog  box  of  the  mystacks  icon.  Oracle 
maintains  security  of  the  database  by  forcing  the  user  to  log  on  each  time  you  enter  the 
database  through  any  application,  so  you  must  log  onto  Oracle  again  at  this  point.  The 
Trace&Transcription  card  then  appears  (Figure  11). 


_ be  j6ig;T&T  diitahase;traceG’tronsci1ption 

Transcription  to  Soar  Matching 


Applications 

(§>  Transcription  to  Soar 
O  Toad  Transcription  into  OB 
O  Load  Soar  trace  into  OB 


Transcription  Query 


Soar  Query 


Standard  Transcription  Query 


Standard  Soar  Query 


The  top  section  of  the  Trace&Transcription  card  allows  access  to  several  related  applic^ons, 
select^  by  clicking  on  the  radio  buttons  to  the  left  of  their  names.  The  first.  Transcription 
to  Soar  Trace  is  the  application  that  allows  the  user  to  link  observed  behaviors  to  Soar 
decision  cycles  (and  is  the  only  one  implemented  in  the  prototype).  The  others.  Load 
Transcriptions  into  DB  and  Load  ^ar  Trace  in  to  DB,  are  future  applications  that 
load  data  directly  horn  the  natural  format  of  the  data  files  rather  than  having  to  put  them  into  a 
format  acceptable  to  Oracle,  as  must  be  done  with  the  prototype  (section  32.1  of  this  report). 
That  is,  these  applications  will  strip  off  standard  file  headers,  provide  correct  indentation, 
and  do  other  formatting  customized  for  the  transcription  and  Soar  trace  files. 

The  middle  section  of  the  Trace&Transcription  card,  when  the  Transcription  to  Soar 
Trace  application  is  selected,  allows  access  to  pre-stored  queries.  Clicking  on  the  boxes 
below  the  Transcription  Query  or  Soar  Trace  Query  headings,  causes  a  dialog  box  to 
appear  so  that  a  different  pre-stoi^  query  can  be  selected.  Making  and  storing  queries  is 
discussed  in  the  next  section  (section  3.2.3). 

The  bottom  section  of  the  Trace&Transcription  card  provides  button  to  Log  On,  Run 
Application  or  Make  Query.  Log  On  allows  another  user  to  log  on,  if  desired.  Make 
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Query  takes  you  to  the  card  shown  in  Figure  16  and  will  be  discussed  in  the  next  section. 
Run  Application  performs  the  queries  previously  selected  and  displays  the  results  of  those 
queries  on  the  Trace& Transcription  Information  card,  shown  in  Figure  12. 


<3 

Trace&Transcription  information 

a  rt««rds  in  sdcettou  1 73 

](  liinr 

D 

•  f*M8ds  in  sdcntiMi  169  | 

tram-cfiptiMi  M  fi 

ifcnniEiaiiin 

snnrM  1 1  1 

IcnrnnE>i 

tfeetsiM 

mrnmb* 

f  Um 

vWW 

cyein 

SMTtrncn 

Olv 

1S70S000 

OK 

s 

0 

6:  gl 

02v 

15706000 

SO 

n 

1 

i:  p9  (top-spoco) 

- 

15707000 

2 

3;  s5 

Odr 

15708000 

so  I  just  «nnt. 

3 

0:  ol2  tbroosing-task) 

15709000 

9 

as>6;  g36  (operator  no-ebango) 

15710000 

5 

2:  p92  (browsing) 

15711000 

6 

3;  s99 

15712000 

7 

(Mv 

15713000 

■bIm  o  Mxk 

8 

15719000 

9 

OSy 

15715000 

I  vAAt  to  &raw. 

10 

3:  s72 

06v 

15716000 

1  ««nt  to  eize. 

11 

0:  o79  (defino-soarcfa.-exi. . 

15717000 

12 

zx>e;  g78  (operator  no-eb. . 

07t 

15718000 

guess  1  «nnt  t. 

13 

2:  p89  (define-seareb-. . 

15719000 

H  to  fiolf . 

19 

3:  s99 

09r 

15720000 

so  lot's  soo 

IS 

0:  olOO  (generate-sear. . 

lOm 

15720000 

C  on  bolp. 

16 

0;  oll2  (evaluate-sear . . 

Xlr 

15721000 

1  wwvt  «  <illo. 

17 

0:  oll8  (generate-sear.. 

n 

I2y 

15722000 

lot's  find. . . 

18 

0;  ol28  (evaluate-sear.. 

ml 

....am. 

M  to  UP  o . 

w 

_ 

w 

Figure  12.  Trace&Transcription  Infonnadon  card. 


The  Trace&Transcription  Information  card  displays  the  results  of  the  selected  queries  with 
the  protocol  transcription  data  appearing  on  the  left-hand  side  and  the  Soar  trace  data 
appearing  on  the  right-hand  side.  Initit^y,  there  are  no  links  between  the  transcription 
records  and  the  trace  records,  so  these  windows  are  not  yoked  and  they  scroll  separately. 


The  function  of  the  Trace&Transcription  application  is  to  allow  an  analyst  to  create  links 
between  the  records  of  the  protocol  transcription  table  representing  individual  observed 
behaviors  and  the  records  of  the  Soar  trace  table  representing  Soar  decision  cycles. 

There  are  two  types  of  links  necessary  to  make  the  relationships  between  these  tables 
apparent  to  the  analyst.  An  observed  behavior  might  have  a  one-to-one  mapping  to  a  Soar 
decision  cycle,  that  is,  the  simulation  of  a  particular  behavior  is  performed  by  a  single  Soar 
operator  and  the  empirical  evidence  for  that  Soar  operator  is  that  particular  operator.  When 
this  relationship  occurs,  then  it  is  often  convenient  to  align  the  behavioral  record  with  the 
Soar  decision  cycle,  allowing  the  analyst  to  see  the  one-to-one  mapping.  This  tj^  of  link, 
called  a  forced_alignment,  is  created  by  clicking  the  mouse  on  the  desired  behavioral  record 
(a  black  circle  will  appear  on  the  far  left  of  the  transcription  line),  then  clicking  on  the  desired 
Soar  decision  cycle  (a  black  circle  will  appear  on  the  far  left  of  Ae  Soar  line),  and  then 
clicking  on  the  Link  button.  The  displays  will  align  accordingly  and  arrowheads  facing  each 
other  will  appear  in  the  center  of  the  screen  indicating  the  link  (Figure  13).  When  a 
forced_alignment  link  is  created,  the  two  screens  become  yoked,  and  scrolling  one  screen 
will  automatically  scroll  the  other  screen. 
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Figure  13.  Example  of  a  forced_alignment  link  (04v  &  dclS).  OSv  and  dcl7  show  the  black  dots 
that  indicate  they  will  also  be  linked  with  a  forced.alignment  link  after  the  Link  process  is  complete. 

The  other  type  of  link  is  needed  when  forced_alignment  would  cause  conflict  in  aligning  the 
data.  This  can  occur  in  two  cases:  wh<>n  linked  behaviors  and  decision  cycles  occur  out  of 
order  (i.e.,  a  behavior  linked  to  a  later  decision  cycle  occurs  earlier  in  time  than  a  behavior 
linked  to  an  earlier  decision  cycle)  and  when  more  than  one  behavioral  record  should  be 
linked  to  a  single  Soar  decision  cycle.  To  create  a  link  that  does  not  force  alignment  on  the 
screen,  hold  the  shift  key  down  when  clicking  on  the  behavior  and  the  decision  cycle 
(dashes  will  ^pear  next  to  the  selected  lines,  rather  than  the  black  circles  in  the 
fbrced.alignment  selections)  and  then  click  on  the  Link  button.  Numbers  will  {q)pear  in  the 
center  of  the  screen  indicating  the  link  (Figure  14).  If  more  than  one  behavior  in  linked  to  a 
single  Soar  decision  cycle,  the  srme  number  will  E^pear  on  each  line  of  the  multi-line 
beluvior  to  indicate  this  many-to-one  link  (Figure  15). 

These  links  are  made  in  the  Trace&Transcription  card  only  until  the  analyst  clicks  on  the 
Update  DB  button.  Then  these  links  are  sent  to  the  POINTERS  table  of  the  database,  and 
will  come  back  into  Trace&Transcription  whenever  the  linked  behaviors  or  decision  cycles 
are  accessed.  It  is  good  practice  to  update  the  database  after  every  few  links  are  creat^. 

The  UnLink  All  button  removes  all  the  links  currently  in  Trace&Transcription,  including 
both  the  ones  permanently  in  the  database  and  those  only  in  the  card  (not  yet  uf^ted).  This 
is  a  powerful  button,  useftil  in  a  prototype  with  only  a  few  links.  However,  it  is  probably 
too  powerful  for  routine  use  in  a  full-fledged  database  and  a  more  local  undo  function  should 
be  ^ded  to  this  card. 
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Figure  14.  Example  of  a  link  without  forced  alignment,  necessary  because  the  linked  behaviors  and 
decision  cycles  are  out  of  mder  with  other  links.  In  diis  example,  decision  cycle  1 14  in  the  Soar  trace 
window  is  linked  to  behavior  number  31v  in  the  transcription  window,  with  the  intervening  links  for 
behavior  32m  and  del  10  and  33v  and  del  12  that  would  conflict  if  this  were  a  forced.alignment  link. 
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Figure  IS.  Example  of  a  many-to-one  link  without  forced  alignment.  Behaviors  V49  and  VSO  are 
linked  to  decision  cycle  143. 
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323  Making  Queries 

Once  the  protocol  transcription  data  are  linked  to  the  Soar  trace  data,  an  analyst  may  want  to 
make  complex  queries  of  the  database  to  explore  these  relationships.  For  instance,  in 
evaluating  the  goodness  of  fit  of  a  cognitive  simulation  like  a  Soar  model,  it  is  desirable  to 
know  how  many  individual  overt  behaviors  are  explicitly  simulated,  and  how  many  aren't. 
More  specific  information  about  the  goodness  of  fit  can  be  had  by  looking  at  what  percentage 
of  verbal  behaviors  are,  and  are  not,  explicitly  simulated,  vs.  the  same  statistics  for  the  motor 
behaviors.  More  qualitative,  but  perhaps  more  useful  in  understanding  the  failings  of  a 
model  is  to  display  those  behaviors  not  simulated  and  some  of  the  behavioral  context  leading 
up  to  each  unsimulated  behavior.  These  questions  can  be  answered  with  this  Oracle  database 
tl^ugh  the  power  of  SQL  queries. 

To  build  a  query  and  store  it  in  the  database  for  repeated  use,  the  analyst  clicks  the  Make 
Query  button  on  the  Trace&Transcription  application  card.  This  action  brings  the  analyst  to 
a  card  with  3  choices;  Construct  New  Query,  Import  Saved  Query  from  Oracle  Query  Tool, 
and  View  &  Modify  Query  in  DB. 

To  create  a  brand  new  query  click  on  the  button  next  to  "Construct  New  Query".  This  brings 
up  the  Oracle’s  Query  Tool  (Chiqiter  3  in  the  Oracle  User's  Guide).  Once  the  query  is  built 
using  the  tool,  then  store  this  query  in  an  ascii  file  by  clicking  on  the  Report  button,  then  on 
the  write-to-disk  icon  on  the  Report  card  (second  one  down  on  the  far  left  of  the  card).^ 
Return  to  the  Make  Query  card  by  going  through  HyperCard  Home,  and 
Trace&Transcription.  The  stored  file  can  then  be  imported  into  the  Trace&Transcription 
application  by  clicking  on  the  button  next  to  "Import  Saved  Query  from  Oracle  Query  Tool" 
(see  the  next  paragraph). 

The  buaon  next  to  "Import  Saved  Query  from  Oracle  Query  Tool"  puts  up  a  dialog  box  that 
requests  the  filename  for  a  previously  created  ascii  query  (see  previous  paragr^h).  Once  a 
file  is  selected,  the  query  is  extracted  and  is  shown  in  the  scrolling  field  (Figure  16).  The 
name  of  the  source  frle  is  displayed  above  the  query.  The  query  can  be  modified  using  all  the 
standard  Macintosh  text  manipulation  techniques.  All  the  changes  to  this  query  are  lo^  to 
this  card,  until  the  Update  DB  button  is  clicked.  First,  enter  die  name  the  query  should  be 
stored  under  in  the  Save.. As...  field  above  the  scrolling  field.  Then  the  Updiate  DB 
button  puts  the  query  in  the  scrolling  field  into  the  Trace&Transcription  database,  in  a 
QUERY  table  ,under  that  name  (it  does  not  change  the  original  ascii  file).  At  any  time,  the 
query  can  be  run  in  the  Trace&Transcription  application  by  clicking  on  the  Run 
Application  button. 

Queries  can  include  either  constants  or  variables.  It  is  often  easier  to  create  a  query  with  a 
particular  example  in  mind,  putting  constants  into  the  query  and  testing  it  with  this  example. 
Then  the  constants  can  be  replaced  by  variables.  Hien,  at  run  time,  the  user  is  asked  for  a 
value  for  each  variable  in  the  query,  making  the  query  much  more  general.  To  make  a 
constant  into  a  variable,  select  the  constant  in  the  scrolling  field  and  click  on  the  Make 
Variable  button.  The  selected  constant  will  then  be  replaced  with  a  variable  of  the  form 
%%VAR$x$%%,  where  x  is  a  number  between  1  and  10,  and  a  dialog  box  will  appear.  The 
dialog  box  will  ask  for  a  name  to  label  the  variable,  to  use  to  ask  for  the  value  of  the  variable 
at  run  time.  The  original  constant  will  become  the  default  value  of  the  variable.  To  make  a 
variable  back  into  a  constant,  select  the  variable  and  click  the  Remove  Variable  button. 
This  replaces  the  variable  with  its  default  value  and  releases  the  variable 


^  Do  not  use  the  Save  Query  button  to  store  queries  because  that  button  saves  them  in  a  format  urueadable  by 
Trace&Transcription. 
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Figure  16.  Import  Saved  Queiy  from  Oracle  Query  Tool  card. 


for  use  in  other  places  in  the  query.  (Trace&Transcription  currently  has  a  10-variable  limit 
per  query,  so  releasing  the  variable  may  be  important.) 


The  last  choice  of  action  on  the  Make  Query  card  is  "View  &  Modify  Query  in  DB".  This 
button  presents  the  same  interface  to  the  user  as  the  "Import  Saved  Query..."  button. 
However,  it  allows  access  to  queries  already  stored  in  the  Trace&Transcription  database, 
rather  than  queries  externally  created  by  the  Oracle  Query  Tool.  As  before,  the  changes  made 
to  the  query  are  only  local  to  the  card  until  Update  DB  is  clicked.  As  before,  the  query  can 
be  run  in  the  Trace&Transcription  ^plication  directly  from  this  card  with  the  Run 
Application  button. 
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